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Abstract 

Psychological studies indicate that emotional states are expressed in the 
way people walk and the human gait is investigated in terms of its ability to 
reveal a person’s emotional state. And Microsoft Kinect is a rapidly developing, 
inexpensive, portable and no-marker motion capture system. This paper gives a 
new referable method to do emotion recognition, by using Microsoft Kinect to do 
gait pattern analysis, which has not been reported. 59 subjects are recruited in 
this study and their gait patterns are record by two Kinect cameras. Significant 
joints selecting, Coordinate system transforming. Slider window gauss hlter. 
Differential operation, and Data segmentation are used in data preprocessing. 
Feature extracting is based on Fourier transformation. By using the NaiveBayes, 
RandomForests, libSVM and SMO classification, the recognition rate of natural 
and unnatural emotions can reach above 70%.It is concluded that using the 
Kinect system can be a new method in recognition of emotions. 
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1. Introduction 

Detection of emotions is generally based on observing facial expressions, lin¬ 
guistic as well as acoustic feature in speech, physiological parameters, gesture 
and body motions[T].Body motions are considered as additional modality to 
estimate the emotion state of a human. Affective body movement provides im¬ 
portant visual cues used to distinguish expression of emotion[2]. Since gait is a 
natural day to day motion, our work concentrates on the recognition of emotions 
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in gait patterns. 

Psychological studies indicate that emotional states are expressed in the way 
people walk and the human gait is investigated in terms of its ability to reveal a 
persons emotional state. In recent years, much research has been done to define 
a normal gait patterns. This task is challenging, because peoples individual 
gait is as unique as their fingerprint [3]. Furthermore, gait is influenced by many 
factors such as age, weight, and possible gait disorders. How such factors or 
combinations of them affect gait is still an open question. 

Our work gives a new referable method to do emotion recognition, by using 
Microsoft Kinect to do gait pattern analysis, which has not been reported. The 
Microsoft Kinect is camera-based sensor primarily used to directly control com¬ 
puter games through body movement. The tracks of the position of the limbs 
and body without the need for handheld controllers or force platforms. Use of 
a depth sensor also allows the Kinect to capture three-dimensional movement 
patterns. The system software enables feature extraction to recognize body joint 
centers. In comparison to conventional motion capture systems, Kinect system 
is low-cost [4], portable,no- marker[5],and easy to use[6]. 

We recruit the subjects and collect the joints timing sequence data by using 
Kinect system, then use machine learning method to do feature extraction and 
classification, and the recognition rate between the natural and unnatural emo¬ 
tions can reach above 70%. 

The remainder of this paper is organized as follow. Section 2 introduces 
the related work of recognition of emotions in walking and the Kinect applica¬ 
tion. Section 3 describes the method of our work, including the experiment, the 
database, and the data processing. Classification and results are presented in 
section 4 and discussed in section 5.The paper ends with a conclusion in section 
6 . 


2 


2. Related work 


In psychology, evidence exists that emotion can be expressed in walking 
and recognized by human observers. In 1987,Montepare et.al’s psychological 
study indicated that observers can identify emotion from variations in walking 
style[7].In 2008,Janssen used the conventional camera to acquire kinetic and 
kinematic data and do recognition of emotions in gait [5] and in 2009-2010,Krag 
did Person-Dependent and Inter-Individual Recognition of emotions by marker- 
based gait analysis using motion capturing system pi [TUI [TTj .But conventional 
3-dimension video-based motion analysis systems allow for comprehensive kine¬ 
matic and kinetic analysis of gait and require large spaces, and considerable 
expertise and are expensive. The marker-based motion capture systems (e.g. 

The VICON system) require precise, tedious and time-consuming maker prepa¬ 
ration, which may affect the subjects emotional states and also expensive. 

Low cost options could include inertial monitoring sensors such as accelerometers [12] 
and gyroscopes, however these sensors possess sources of error such as signal drift 
and noise which impede their accuracy |13|. 

Microsoft Kinect is a rapidly developing, inexpensive, portable and no¬ 
marker motion capture system. Early reports suggests the Kinect can identify 
pose[T3| and simple stepping movements[T^ in healthy adults. Recently, clinical 
researchers have reported interesting applications using the Kinect system, for 
example, an interactive game-based rehabilitation tool for balance training[16j 
and and a 3-D body scanner |T7| .And methods have been proposed to detect gait 
patterns in walking data obtained with Kinect: In 2014,Auvinet used a Kinect 
to detect the gait cycles in treadmill[TH]; Yeung evaluated the Kinect as a clinical 
assessment tool of body sway[T9];Galna used Kinect to measure the movement 
in people with Parkinsons disease ^D]. 

Previous studies have validated the Kinect as a motion capture system. Ac¬ 
curacy and sensitivity of kinematic measurements obtained from Kinect, such 
as reaching distance, joint angles, and spatial-temporal gait parameters, were 
comparable to a VICON svstempT] .Evidence exists that it can accurately assess 
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the gait patterns dynamics during walking|22j. 

However, the use of Kinect systems as a emotional recognition tool has not 
been reported. 

3. Methods 

3.1. Experiment 

59 healthy young subjects(32 females and 27 males) from University of Chi¬ 
nese Academy of Sciences participated in this study. They reported no injuries, 
illnesses or other condition influence their gait patterns. There is a 6 meters 
long footpath in the experiment environment, and at the two sides of the foot¬ 
path is two Kinect camera. The experiment environment is shown in Figure 1 
and Figure 2 

Every subject is required to walk around the footpath for 2 minutes, then 
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Figure 1: the description of the experiment environment 



they evaluate their own emotional state(angry or happy) and give a score(l to 
10). Then a stimulus video for one emotion was shown to participants. When 
they finish watching, they walk around the footpath for 1 minute. Every sub¬ 
ject’s gait patterns is record by two kinect camera. When they finish walking, 
they give two scores(l to 10) to evaluate their own emotion state finishing watch¬ 
ing the video and finishing the second walking. 

The experiment is held twice. In the first round of experiment, subjects walk 
with natural and angry emotion. In the second round of experiment, subjects 
walk with natural and happy emotion. 
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Figure 2: the scene of the experiment environment 


3.2. Database 

The Kinect system is placed at the two sides of the footpath, 30Hz video data 
are acquired from every Kinect camera using the official Microsoft software de¬ 
velopment SDK Beta2 version and customized software(Microsoft Visual Studio 
2012),One frame data contains the 3-dimensional position of 25 joints over time, 
which affords a 75 dimensional vector. The 25 joints include head, shoulders, 
elbows, wrists, hands, spine ,hips, knees, ankles and feet as shown in Figure 3. 

The 3D coordinate system of Kinect use the Kinect camera as the origin 



Figure 3: Stick figure and location of body joint centers estimated by Kinect 
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and the unit of 3 dimension is meter. The 3D coordinate system of Kinect is 
shown in Figure 4. 
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Figure 4: Kinect 3D coordinate 


3.3. Data processing 
3.3.1. Preprocessing 

Signifieant joints seleeting. According to sport anatomy theory, some joints 
don’t change much when people walk, So we select 14 significant joints ,which 
include spinebase, neck, shoulders, wrists ,elbows, hips , knees ,and ankles ,to 
analyze gait patterns, then one frame data contains the 3-dimmension position 
of 14 significant joints, which affords a 42 dimension vector: 


jt = [a;i,j/i, 2 :i,X 2 , 2 / 2 , 22 ,-••,2:14,2/14,2:14] 


( 1 ) 


Suppose the process of one subject walk around the footpath one time(under 
an emotional state) consists of T frames, called one walk, the complete dataset 
of one walk is described by the matrix: 


Jt = [ji,j2, • • •,it, • • • 


( 2 ) 
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Coordinate system transforming. Considering the different people have different 
position relative to Kinect camera when walk on the footpath, so using Kinect 
coordinate system has error in gait pattern analysis. To eliminate the error , we 
change the coordinate system by use the position of spinebase in every frame of 
data as the origin. 

In the vector ,the first three columns, ,are the coordinates of spinebase joint, 
so the coordinate transform is given by: 


X, = x, — Xi 


X, = X,- — Xi 


= x\- x\ 


(1 < t < T, 2 < i < 14) 


(3) 


Slider window gauss filter. The gait data set of every walk has noises and burrs, 
To smooth the data set, we apply slider window gauss filter to every column 
of matrix J, the length of the window is 5 and the convolution kernel c = 
[1,4,6,4,1]/16 

Then the process of the filtering is given by: 
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(1 < t < T, 1 < i < 14) 


Differential operation. Considering the change of joints’ position between each 
frame can reflect the people’s gait patterns more well than the joints’ position 
itself, so we apply the differential operation on the matrix J. 

The differential operation is given by: 


jt-i = jt - jt-i(2 <t<T) 


(5) 


Data segmentation. Considering in the process of one subject walk around the 
footpath one time(one walk) have several straight walk segment, and the joints 
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coordinate acquired by Kinect is not accurate when people turn round on the 
side of footpath. So we cut the complete dataset of one walk into several straight 
walk segments. Some segments record the gait patterns when subjects face to 
the camera, called front segments, the other segments record the gait patterns 
when subjects are back to the camera, called back segments. In order to ensure 
each segment covers at least one stride, we only choose the segment which 
contains at least 40 frames to analyze. 

Suppose one walk J contains n front segments and m back segments, which 
is described by a series of matrices: 


{ Fronti, 1 < i < n 
Backj, 1 < j < m 


( 6 ) 


3.3.2. Feature Extraction 

The gait patterns between the front segment and the back segment is pretty 
different, so we extract features from front segments and back segments sepa¬ 
rately. 

From what has been presented above, the process of one subject walk around 
the footpath one time (with an emotion state),called one walk, described by a 
matrix J, contains n front segments and m back segments. 

First, we extract features from front segments. Human walking is periodic, 
and each segment covers at least one stride; therefore, Fourier transformation 
is applied to model the behavior of the each front segment Fronti,Do Fourier 
transformation on each column of Fronti,The main frequency 
and the corresponding phase ip\,Lp\,..., Lp\ 2 , are extracted.Because of one walk 
contains n front segments and different processes of walk has different number 
of segments. We select mean features of every front segment. Then get a 84 
dimension vector: 


1 

n 


[/l) /2 ) ■ ■ • > 1X2 t‘P\j‘P2t ■ ■ ^ ‘^^ 42 ] 


Featurcfront 


(7) 



Second, extract features from back segments in same way, get another 84 
dimension vector: 

Featuretack = —Y][fl,f 2 , ■ ■ ■, ■ ■ ■, <^ 42 ] (8) 

i 

Combine Feature front and Featurcback to get a 168 dimension vector: 

Feature = [Featurefront, Featuretack] (9) 

The vector Feature describe the process of one subject walk on the footpath 
one time(with an emotional state), and the classification base on these features. 

4. Classification and Result 

For recognition, several standard classifiers are compared. NaiveBayes, Ran¬ 
dom Forests , LibSVM and SMO classifiers are used for classification. And the 
recognition rate is calculated using 10-fold cross validation. 

Table 1 lists the accuracy of each classifier to recognize the natural emotion 
and angry emotion , the gait datasets are collected by one Kinect camera ,called 
KINECTl, in first round of experiment. 

Table 2 lists the accuracy of each classifier to recognize the natural emotion 


Table 1: the accuracy of recognition of natural and angry emotion, The gait datasets are 
collected by KINECTl_ 


Classifier 

NaiveBayes 

RandomForests 

LibSVM 

SMO 

Accuracy(%) 

77.5862 

53.4483 

64.6552 

56.0345 


and angry emotion , the gait datasets are collected by the other Kinect camera, 
called KINECT2, in first round of experiment. 

Table 3 and Table 4 list the accuracy of each classifier to recognize the nat¬ 
ural emotion and happy emotion , the gait datasets are collected by KINECTl 
and KINECT2, in second round of experiment. 

Table 5 and Table 6 list the accuracy of each classifier to recognize the an- 
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Table 2: the accuracy of recognition of natural and angry emotion, The gait datasets are 
collected by KINECT2_ 


Classifier 

NaiveBayes 

RandomForests 

LibSVM 

SMO 

Accuracy(%) 

72.0339 

53.3898 

72.0339 

54.2373 


Table 3: the accuracy of recognition of natural and happy emotion, The gait datasets are 
collected by KINECTl_ 


Classifier 

NaiveBayes 

RandomForests 

LibSVM SMO 

Accuracy(%) 

71.1846 

50.8475 

58.4746 


gry emotion and happy emotion , the gait datasets are collected by KINECTl 
and KINECT2, in first and second round of experiment. 


5. Discussion 

Recognition of emotions in gait patterns is challenge from data mining. Ex¬ 
pression of emotions varies within subjects, ground truth of labeled data is not 
assured in effective computing and also gait patterns are highly individual. But 
the gait is capable to give cues about ones emotion state. 

In comparison to marked-based system, Kinect may be inaccuracy in joint 
center estimation, it could be a possible cause for a offset error. 

From the result we can see that the recognition rate between natural and 
unnatural emotion can reach more than 70%,But the recognition rate between 
different unnatural emotions (angry and happy) is not very good. It shows that 
people’s gait may be similar with angry and happy emotional states. 

Within significant joints selecting approach for data preprocessing,results 

Table 4: the accuracy of recognition of natural and happy emotion. The gait datasets are 
collect ed by KINECT2 _ 


Classifier 

NaiveBayes 

RandomForests LibSVM SMO 

Accuracy(%) 

60.1695 

53.3898 
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Table 5: the accuracy of recognition of angry and happy emotion, The gait datasets are 

collec ted by KINECTl _ 

Classifier NaiveBayes RandomForests LibSVM SMO 

Accuracy(%) 55.0847 54.2373 62.7119 54.2373 


Table 6: the accuracy of recognition of angry and happy emotion, The gait datasets are 

collec ted by KINECT2 _ 

Classifier NaiveBayes RandomForests LibSVM SMO 

Accuracy(%) - 54.5423 - 50.0000 


shows good performance regarding accuracy, Using the whole 25 joints position 
leads to lower recognition rates, which can not reach 70%. 

6. Conclusion and Future Work 

This work gives a new referable method to do emotion recognition, by using 
Kinect system to acquire gait datasets and do feature extraction and classifi¬ 
cation. The natural and unnatural emotions are in general better recognizable 
in gait patterns, but the recognition of angry and happy emotions is not very 
good. And due to low number of subjects, the recognition rate for emotions in 
walking remains around chance level in. So, further improvement is required. 
This includes more gait patterns data acquisition by recruit more subjects and 
further improvement of feature extraction as well as classification. 
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